EVALUATING DIGITAL LIBRARIES: A SOCIOTECHNICAL APPROACH [Work-in-progress: Please do not cite]

The National Science Digital Library (NSDL: http://www.nsdl.org) is an NSF-funded program that supports the creation and dissemination of digital educational resources for all levels of science, technology, engineering, and mathematics education. NSDL is conducting a programwide evaluation of all its activities since the program’s inception in 2000, with the aim of inventorying achievements to date, and identifying directions for future development. The scale, complexity and distributed nature of the NSDL program pose significant challenges for this evaluation work. This poster outlines a sociotechnical theoretical framework, the ‘resource lifecycle,’ that is being used to guide the evaluation of the NSDL program, and to represent evaluation findings to internal community members and to external partners such as NSF. The poster describes the theoretical underpinnings of the model and outlines some of the early evaluation results obtained from using the framework. The National Science Digital Library The National Science Digital Library (NSDL: nsdl.org) is an NSF-funded digital library program that provides online access to high quality educational resources for all levels of science, technology, engineering, and mathematics education. Since 2000 NSDL has funded over 200 projects in a wide range of research areas, including library architecture, database and search engine design, web site design and usability, resource creation, metadata standards, collection development, and community and outreach activities. The outcomes of these individual projects are expected to integrate with and contribute to NSDL infrastructure, and to support the operations of the NSDL program as a whole. NSDL is now conducting a program-wide evaluation of these projects, and of the communication and knowledge structures that support and integrate NSDL as an organization. The evaluation is summative, including an inventory of NSDL activities and achievements to date, and also formative, working towards identifying future strategies for NSDL development, for instance as a component of cyberinfrastructure. There are significant challenges associated with evaluating NSDL. These challenges arise from the distributed and organizationally heterogeneous nature of the NSDL program, which has developed a wide range of collections, services, and technological and social architectures; has involved researchers from a wide range of disciplines and ‘communities of practice’ (Wenger, 1999), including information and library scientists, computer scientists, pedagogists, and practicing educators; is used by a wide range of audiences in a wide variety of local contexts; and has been composed of projects distributed across the USA, a number of which are no longer active, making access to project documentation difficult. The highly complex structure of NSDL raises a number of evaluation questions, including: Scope of the inquiry: What program activities should an NSDL evaluation focus on? Current NSDL activities? All NSDL activities? If specific activities are selected, by what criteria should they be chosen? Granularity of the inquiry: Should an evaluation examine all the individual projects within NSDL? Should it focus on the organizational processes whereby NSDL coordinates its practices? Should it focus on NSDL projects, or on NSDL as a program? Representation of the inquiry: Conceptual coherence is crucial for the presentation of evaluation results to internal and external partners; how may NSDL’s progress towards its goals, and the lessons learned over the past years, best be communicated to its members and sponsors? In other words: Where should an evaluation of NSDL begin, and what sort of questions should it ask? A Sociotechnical Model for Digital Library Evaluation Successful digital library evaluation depends on the identification of a shortlist of ‘doable’ evaluation questions that address both on-the-ground practicalities (such as access to data and limited evaluation resources) and long-term strategic concerns (such as possible directions for long-term growth) (Reeves et al., 2003). It also requires a ‘multi-faceted’ approach that takes into account the various contexts in which digital library development and use occur (e.g. Marchionini et al., 2003; Van House et al., 2003). It should also be capable of addressing the concerns of a range of stakeholders, including the library’s users, sponsors, and developers. Given the large range of possible variables and relationships involved in digital library use, the identification of ‘doable’ and multifaceted evaluation questions is made easier if this work is carried out within a coherent theoretical framework, that is capable of describing, organizing and contextualizing the range of variables in the evaluation site in question. In the case of NSDL, the evaluation is being informed by a sociotechnical theoretical approach. Briefly, sociotechnical theory recognizes that technology impacts peoples’ practices, and also that people and their practices – including their local cultural practices – shape the ways in which technologies are adapted and used (e.g. Bijker et al., 1987; Bijker, 1995). This mutually constitutive relationship between technologies and practices produces sociotechnical systems that are complex and emergent. A sociotechnical approach to evaluating NSDL therefore models NSDL as a complex mixture of technologies, people, practices, institutions, cultural frameworks, and other phenomena, that are connected in mutually constitutive and evolving (i.e. in complex systemic) ways. The sociotechnical model should also identify a unit of analysis that can be ‘followed’ through (and thus provide a coherent perspective on) the sociotechnical system. The specific unit of analysis developed for the evaluation of NSDL is the digital resource. This unit of analysis assumes that the overall purpose of the NSDL program is to transform scientific data into valuable pedagogical resources, for use in a range of educational settings (Khoo, 2005). This transformation is achieved within the NSDL program through a series of value-adding operations, that are coordinated amongst individual projects. For instance, a project may create digital educational resources; or it may review such resources for quality; or it may add metadata to a resource; or it may aggregate resources into a collection; or it may make resources discoverable through a search interface; or it may undertake outreach activities to make teachers aware of its resources; and so on. Each of these activities requires specific sets of skills and practices; and these skills and practices are coordinated in a longitudinal workflow (that may be likened to a production-line model) in which the output of each stage forms the input of the succeeding stage, and in which each activity incrementally adds value and utility to the resource (note that NSDL projects may contribute to more than one stage of this workflow). For example, a resource that has been reviewed and revised, is more valuable than a resource which has not; a resource described with metadata, is more valuable than a resource which is not so described; a resource that is discoverable by a search engine is more valuable than a resource that is not discoverable; a resource in a library that practices outreach has more value than a resource that is in a library no-one has heard of; and so on. In the case of NSDL, a total of 10 interdependent value-adding stages have been identified; these range from resource creation, through the addition of itemand collection-level metadata to resources, to resource use and reuse (figure 1). Taken together, these stages constitute a ‘lifecycle’ for educational digital resources. Note that this cycle can be iterative, in that a resource that is the product of one cycle, can serve as the basis for the development of a new resource in a following cycle. Figure 1. Resource lifecycle model (detailed version) One implicit and important dimension of the resource lifecycle not represented in figure 1 is that of the NSDL program’s organizational communication and knowledge processes, which play a crucial role in coordinating and supporting the efforts of individual NSDL projects. The central role of such communication and knowledge is represented in figure 2, which aggregates the ten value-adding stages identified in figure 1 into four core NSDL value-adding activities: (1) resource creation and review; (2) resource aggregation and collection development; (3) web site and search engine design; (4) classroom use/reuse and educator tools. In the NSDL program, and in figure 2, these four core activities are coordinated through a fifth core activity, NSDL’s communication and knowledge infrastructure, which includes such processes as meetings and workshops, committees, e-mail lists, wikis, outreach activities, etc. Figure 2. Resource lifecycle model (overview) Each of the NSDL project activities identified in these figures is expected, in one way or another, to add value to the resources in NSDL, and thus to the overall value of NSDL as a library. For instance, a resource that is reviewed for pedagogical effectiveness, scientific accuracy, and technological functionality, will be of a higher quality and will also be more valuable than a resource that is not, and will contribute more value to NSDL than a resource that is not so reviewed. Similarly, a resource described by accurate metadata is more valuable for NSDL than a resource that is not; a resource discoverable by a powerful search and discovery tool is more valuable than one that is not; and so on. While the activities of an individual NSDL project may not directly cover all the stages of the resource lifecycle, all stages of the resource lifecycle should be supported by NSDL project activities in one form or another. When conceived of as value-adding activities with specific desired outcomes, each of the resource lifecycle stages identified in the above figures thus provides a locale and range of potential variables for specific sociotechnical evaluation activities. This is because each stage is supposed to add value to NSDL’s educational resources; and as each stage also involves the interaction of (a) people, disciplines and communities of practice (e.g. librarians), with (b) particular technologies (e.g. cataloging tools), in order to generate (c) particular outcomes (e.g. metadata of a certain quality), unpacking the relationships between these variables (communities of practice, technologies, outcomes, etc.), in each of the stages, and reviewing the contribution of these processes to designated value-added outcomes, provides a context within which evaluation models – such as the assessment of projects’ resource creation processes and resource quality, or a webmetrics analyses of web site traffic, or user interface testing of projects’ web sites, search engines, etc. – can take place. For example, stages 1 and 2 in figure 1 refer to the processes of ‘creation’ and ‘review’; and possible evaluation activities here include assessment of the quality of the resources created, assessment of the rubrics projects use to create resources, and assessment of the support that NSDL provides individual projects to carry out the tasks of resource creation and review. Again, evaluation of stage 7, ‘search and discovery,’ could include the evaluation of such factors as the quality of a collection’s metadata (which dictates what resources are discoverable in the first place), and the ease-of-use of the web-based search interface. Examples of NSDL Evaluation Activities Evaluation activities have been initiated in all of the core activity areas identified in figure 2 (including assessments of the efficacy of NSDL’s organizational communication and knowledge processes). A summary of these activities follows (full reports of these activities are on the NSDL evaluation wiki: http://eval.comm.nsdl.org/). Resource creation and review (core activity 1) Resource creation and review processes in NSDL projects were evaluated in a web survey e-mailed to all past and present NSDL projects. Responses suggested that the majority of projects did follow resource creation guidelines, and some projects supplied copies of their guidelines, to be made available as models for other projects within NSDL (these outlines have been published on the NSDL evaluation wiki). The presence of these formal guidelines suggests in turn that the resources created by these projects are themselves of a guaranteed quality. One unanticipated outcome of this survey that many respondents ranked NSDL’s organizational communication infrastructure (e-mail lists, wikis, etc.) as of low importance; instead, respondents referred to their preference for meeting with other project members face-to-face, and to having personal contacts in these projects. Collection development and metadata (core activity 2) Metadata quality is an important component of the resource lifecycle model, with better quality metadata supporting better search results for users. Initial analyses are being conducted with the NSDL Metadata Repository (MR), and the soon-to-be-launched NSDL Data Repository (NDR), to assess such factors as field completion rates, and the quality of data in specific fields. The results of these analyses will be used to inform search engine and search page design. Search page usability (core activity 3) User-testing of the search functions of the nsdl.org search pages is being carried out. This has included preliminary user testing, interviews with educators, and paper prototyping of a new search results page design. Early findings suggest that many educators still rely heavily on subtle analyses of the textual description of each resource to sort their search results, and are (currently) less interested in using sophisticated advanced search/filtering features to get the results they want. However, the ways in which this behaviour plays out varies according to the grade-levels taught by the educator; for instance, university student teaching assistants are more interested in specific subject searches than are K-12 teachers. The results of this testing will inform more comprehensive user-testing by internal and external HCI specialists later in 2006. Webmetrics (core activity 4) Individual NSDL projects often carry out their own web metrics analyses, but historically there has been little standardization in tools or reports across the NSLD program. NSDL has therefore contracted standardized cross-project third-party web metrics from Omniture (omniture.com), in order to identify typical patterns of use across NSDL, and to inform future web site design (for instance by identifying heavilyand lightly-used sectors of the NSDL web site). Initial evaluation efforts here have been aimed at identifying consistent crossproject metrics that can demonstrate the impact of NSDL. Project-level evaluation practices (core activities 1-4) An online survey designed to collect individual projects’ own evaluation activities indicated that while projects are eager to carry out this work, they often face obstacles based on lack of funds, time, and expertise. This suggests the need for some kind of centralized evaluation service that could be provided by NSDL, which could provide contract evaluation services for individual projects. Annual Meeting and workshops (core activity 5) NSDL activities such as the NSDL Annual Meeting and workshops help to support community and communication amongst NSDL projects, and to provide opportunities for new collaborations and innovation. Paper and online surveys conducted after the 2005 Annual Meeting suggested that NSDL projects strongly value opportunities to meet and interact face-to-face, and to develop their knowledge and possibilities for collaboration. These findings replicate the findings of the other NSDL surveys mentioned above, which similarly have revealed the importance of personal networks and face-to-face communication to individual members of individual NSDL projects.